The quantity and diversity of informational content accessible on data networks, is increasing constantly and at a substantial rate. This development creates difficulties locating specific information content. Same difficulties arise when locating specific information content in multimedia streams, television broadcasts and radio broadcasts. Various network-based sites maintain generalized content based searching. Another problem involves the size of the collected indexes. Traditional search engines typically perform a consecutive scan of the information sources and create substantial indexes that can be later searched in response to a query. The indexes generated in this manner are usually very large on the order of many gigabytes.
Another significant disadvantage of the conventional search engines relates to the need to ensure the timeliness of the information maintained in the indexes. There is a need to provide a search engine on data networks that is characterized by a fast response and high update rate of the index, both in relation to an addition of content to the index and in relation to a deletion of content from said index. For large collections, the indexes need to be rebuilt to add or remove information sources-terms relations. The process of building and rebuilding an index involves considerable time delay as known index formation processes are one order of a magnitude slower than the typical rate of source content accumulation rate.
Prior art search engines are optimized to huge amounts of data. Therefore, great effort is invested minimizing index sizes in order to keep as much index data as possible in the fast RAM (Read Only Memory) storage instead of being kept on external storage devices. As a result, the indexing process as well as index data deletion are complicated and slow processes.
Therefore, there is an urgent need for a search engine that is fast, accurate, scalable without significant loss of performance and can be maintained to be significantly current in real-time. Especially there is a need to provide such a search engine that is further adapted to perform a complex search in real time.